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Abstract. We analyze stability of consensus algorithms in networks of multi-agents 
with time-varying topologies and delays. The topology and delays are modeled as induced 
by an adapted process and are rather general, including i.i.d. topology processes, asynchro- 
nous consensus algorithms, and Markovian jumping switching. In case the self-links are 
instantaneous, we prove that the network reaches consensus for all bounded delays if the 
graph corresponding to the conditional expectation of the coupling matrix sum across a 
finite time interval has a spanning tree almost surely. Moreover, when self-links are also 
delayed and when the delays satisfy certain integer patterns, we observe and prove that the 
algoiithm may not reach consensus but instead synchi'onize at a periodic trajectory, whose 
period depends on the delay pattern. We also give a brief discussion on the dynamics in the 
absence of self-links. 

1 . Introduction. Consensus problems have been recognized as important in distribution 
coordination of dynamic agent systems, which is widely applied in distributed computing 
[21], management science [5], flocking/swarming theory [32], distributed control [10], and 
sensor networks [26]. In these applications, the multi-agent systems need to agree on a 
common value for a certain quantity of interest that depends on the states of the interests 
of all agents or is a preassigned value. The interaction rule for each agent specifying the 
information communication between itself and its neighborhood is called the consensus 
protocol/algorithm. A related concept of consensus, namely synchronization, is considered 
as "coherence of different processes", and is a widely existing phenomenon in physics and 
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biology. Synchronization of interacting systems has been one of the focal points in many 
research and application fields [33, 16, 29]. For more details on consensus and the relation 
between consensus and synchronization, the reader is referred to the survey paper [27] and 
the references therein. 

A basic idea to solve the consensus problem is updating the current state of each agent 
by averaging the previous states of its neighborhood and its own. The question then is 
whether or under which circumstances the multi-agent system can reach consensus by the 
proposed algorithm. In the past decade, the stability analysis of consensus algorithms has 
attracted much attention in control theory and mathematics [27]. The core purpose of 
stability analysis is not only to obtain the algebraic conditions for consensus, but also to get 
the consensus properties of the topology of the network. The basic discrete-time consensus 
algorithm can be formulated as follows: 

=xl+eJ2 (a;*- - xl), ^ = 1, . . . , m, (1) 

where .t* G R denotes the state variable of the agent i, t is the discrete-time, Ni denotes 
the neighborhood of the agent i, and e is the coupling strength. Define C = [UjYP'j^i as the 
Laplacian of the graph of the network in the manner that lij ^ 1 if i ^ j and a link from j 
to i exists, lij = if that i ^ j and no link from j to i exists, and la = — ^ij- With 
G = I — e£, (1) can be rewritten as 

a;*+i = Gx\ (2) 

where ^] . If the diagonal elements in G are nonnegative, i.e., < e < 

1/ max; la, then G is a stochastic matrix. Eq. (2) is a general model of the synchronous 
consensus algorithm on a network with fixed topology. The network can be a directed 
graph, for example, the leader-follower structure [22], and may have weights. 

In many real-world applications, the connection structure may change in time, for in- 
stance when the agents are moving in physical space. One must then consider time-varying 
topologies under link failure or creation. The asynchronous consensus algorithm also indi- 
cates that the updating rule varies in time [9]. Thus, the consensus algorithm becomes 

a;*+i = G{t)x\ (3) 

where the time-varying coupling matrix G{t) expresses to the time-varying topology. We 
associate G{t) with a directed graph at time t (see Sec. 2), in which Gij [t] > implies 
that there is a link from j to i at time t, which may be a self-link if i = j. Note that the 
self links in G arise from the presence of the Xi on the right hand side of (1); they do not 
necessarily mean that the physical network of multi-agents have self-loops. 

Furthermore, delays occur inevitably due to limited information transmission speed. 
The consensus algorithm with transmission delays can be described as 

where r*^ G N, i, j = 1, . . . , m, denotes the time-dependent delay from vertex j to i. A 
link from j to i is called instantaneous if r*j = Vt, and delayed otherwise. 

In this paper, we study a general consensus problem in networks with time-varying 
topologies and time delays described by 

m 

=J2G,,{a*)x'f''^^^'\ z=l,...,m, (5) 
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as well as the more general form 

^*"''=EE^^('^*)<^ * = l,...,m. (6) 

Note that (5) can be put into the form (6) by partitioning the inter-links according to delays, 
where tm is the maximum delay. However, (6) is more general, as it in principle allows for 
multiple links with different delays between the same pair of vertices. In particular, there 
may exist both instantaneous and delayed self-links, which may naturally arise in a model 
like (1) where the term Xi appears both by itself as well as under the summation sign. In 
reference to (6), we talk about self-link(s) when GJ", ^ 0, which may be instantaneous 
or delayed depending on whether t = or t > 0, respectively. In equations (5)-(6), cr* 
denotes a stochastic process, G(cr*) = [Gij(cr*)]"j-^j = Er=o ^'iji^*)Xi,j=i ^ stochas- 
tic matrix, Tij{a*) G N is the stochastically-varying transmission delay from agent j to 
agent i. This model can describe, for instance, communications between randomly moving 
agents, where the current locations of the agents, and hence the links between them, are 
regarded as stochastic. Furthermore, the delays are also stochastic since they arise due to 
the distances between agents. In this paper, {cr*} is assumed to be an adapted stochastic 
process. 

Deflnition 1.1. (Adapted process) Let {Ak} be a stochastic process defined on the basic 
probability space {fl, P}, with the state space 51, the cr-algebra and the probability 
P. Let {J-'^} be a filtration, i.e., a sequence of nondecreasing sub-cr-algebras of If A^ 
is measurable with respect to (w.rt.) 7^'"', then the sequence {A^, J^'^} is called an adapted 
process. 

Via a standard transformation, any stochastic process can be regarded as an adapted 
process. Let {^t} be a stochastic process in probability spaces {51*, H*, P*}. Define il = 
Ylt 51*, J' and P are both induced by J^^ and Yit IP', where Y[ stands for the Cartesian 
product. Let cr* = [(,'']k=i and J^* be the minimal cr-algebra induced by 111=1 Then 
J^* is a filtration. Thus, it is clear that the notion of an adapted process is rather general, 
and it contains i.i.d. processes, Markov chains, and so on, as special cases. 

Related work. Many recent papers address the stability analysis of consensus in net- 
works of multi-agents. However, the model (5) with delays we have proposed above is 
more general than the existing models in the literature. We first mention some papers where 
models of the form (3) are treated. A result from [25] shows that (3) can reach consensus 
uniformly if and only if there exists T > such that the union graph across any T-length 
time period has a spanning tree. Ref. [2] derived a similar condition for reaching a consen- 
sus via an equivalent concept: strongly rooted graph. Our previous papers [19, 20] studied 
synchronization of nonlinear dynamical systems of networks with time-varying topologies 
by a similar method. Ref. [36] has pointed out that under the assumption that self-links al- 
ways exist and are instantaneous (i.e. without delays), the condition presented in Ref. [25] 
also guarantees consensus with arbitrary bounded multiple delays. However, this crite- 
rion may not work when the time-varying topology involves randomness, because for any 
T > 0, it might occur with positive probability that the union graph across some T-length 
time period does not have a spanning tree for any T. Refs. [14, 35, 31] studied the con- 
sensus in networks under the circumstance that the processes {G(t)}t>o are independently 
and identically distributed (i.i.d.) and [38] also investigated the stability of consensus of 
multi-agent systems with Markovian switching topology with finite states. In these papers, 
consensus is considered in the almost sure sense. Ref. [8] studied a particular situation with 
packet drop communication. The most related literature to the current paper is [18], where 
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a general stochastic process, an adapted process, was introduced to model the switching 
topology, which generalized the existing works including i.i.d. and Markovian jumping 
topologies as special cases. The authors proved that, if the 5-graph (see its definition in 
Sec. 2.2) corresponding to the conditional expectation of the coupling matrix sum across 
a finite time interval has a spanning tree almost surely, then the system reaches consensus. 
However, none of those works considered the stochastic delays but rather assumed that 
self-links always exist. There are also many papers concerned with the continuous-time 
consensus algorithm on networks of agents with time-varying topologies or delays. See 
Ref. [28] for a framework and Ref. [27] for a survey, as well as Refs. [24, 1, 37, 23], 
among others. Also, there are papers concerned with nonlinear coupling functions [6] and 
general coordination [17]. 

Statement of contributions. In the following sections, we study the stability of the 
consensus of the delayed system (5), where cr* is an adapted process. First, we consider the 
case that each agent contains an instantaneous self-link. In this case, we show that the same 
conditions enabling the consensus of algorithms without transmission delays, as mentioned 
in Ref. [18], can also guarantee consensus for the case of arbitrary bounded delays. Second, 
in case that delays also occur at the self-links (for example, when it costs time for each agent 
to process its own information), and only certain delay patterns can occur, we show that the 
algorithm does not necessarily reach consensus but may synchronize to a periodic trajectory 
instead. As we show, the period of the synchronized state depends on the possible delay 
patterns. Finally, we briefly study the situation without self-links, and present consensus 
conditions based on the graph topology and the product of coupling matrices. 

The basic tools we use are theorems about product of stochastic matrices and the results 
from probability theory. Ref. [3] has proved a necessary and sufficient condition for the 
convergence of infinite stochastic matrix products, which involves the concept of scram- 
blingness. Ref. [34] provided a means to get scrambling matrices (defined in Sec. 2.2) 
from products of finite stochastic indecomposable aperiodic (SIA) matrices and Ref. [36] 
showed that an SIA matrix can be guaranteed if the corresponding graph has a spanning 
tree and one of the roots has a self-link. The Borel-Cantelli lemma [7] indicates that if the 
conditional probability of the occurrence of SIA matrices in a product of stochastic matri- 
ces is always positive, then it occurs infinitely often. These previous results give a bridge 
connecting the properties of stochastic matrices, graph topologies, and probability theory 
which we will call upon in the present paper. 

The paper is organized as follows. Introductory notations, definitions, and lemmas are 
given in Sec. 2. The dynamics of the consensus algorithms in networks of multi-agents 
with switching topologies and delays, which are modeled as adapted processes, are studied 
in Sec. 3. Applications of the results are provided in Sec. 4 to i.i.d. and Markovian jumping 
switching. Proofs of theorem are presented in Sec. 5. Conclusions are drawn in Sec. 6. 

2. Preliminaries. This paper is written in terms of stochastic process and algebraic graph 
theory. For the reader's convenience, we present some necessary notations, definitions 
and lemmas in this section. In what follows, N_ denotes the integers from 1 to N, i.e., 
N = {1, . . . ,N}. For a vector v = [vi, . . . ,w„]^ £ M", ||v|| denotes some norm to 
be specified, for instance, the norm |[t;||i = k'il- ^ denotes the set of positive 

integers and Z denotes the integers. For two integers i and j, we denote by {i)j the quotient 
integer set {kj + i : fc G Z}. The greatest common divisor of the integers ii, . . . , i/^ is 
denoted gcd(ii , . . . , ) . The product nA:=i °f matrices denotes the left matrix product 
Bn X • • ■ X Bi. For a matrix A, Ajj or [A]ij denotes the entry of A on the ith row and 
jth column. In a block matrix B, Bij or [B]ij can also stand for its i, j-th block. For 
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two matrices A, B of the same dimension, A > B means Aij > Bij for all i, j, and the 
relations A > B, A < B, and A < B are defined similarly. denotes the identity matrix 
of dimension m. 

2. 1 . Probability theory. {51, J', P} is our general notation for a probability space, which 
may be different in different contexts. In this notation, fl stands for the state space, T 
the Borel cr-algebra, and P{-} the probability on ft. Ep{-} is the expectation with respect 
to P (sometimes E for simplicity, if no ambiguity arises). For any cr-algebra Q ^ 
E{-|C?} (P{-|^?}) is the conditional expectation (probability, respectively) with respect to Q. 
It should be noted that both E{-|C/} and P{-|^?} are actually random variables measurable 
w.rt. Q. The following lemma provides the general statement of the principle of large 
numbers. 

Lemma 2.1. [7] {The Second Borel-Cantelli Lemma) Let Tn, n > be a filtration with 
J-'o = {0, 51} and Cn, n>l a sequence of events with C„ G J-n- Then 

+ CXD 

{C„ infinitely often } = { ^ P{C n\^n-i \ = +00} 

n=l 

with a probability 1, where "infinitely often" means that an infinite number of {C„}5^]^ 
occur 

2.2. Stochastic matrices and graphs. An m x m matrix A = [fly]™ ^1 is said to be a 
stochastic matrix if > for all i, j = 1, . . . , m and X^Jli CLij = 1 for all i = 1, . . . , to. 
A matrix A G M™ ™ is said to be SIA if A is stochastic, indecomposable, and aperiodic, i.e., 
lim„^oo A"" converges to a matrix with identical rows. The Hajnal diameter is introduced 
in Ref. [12, 13] to describe the compression rate of a stochastic matrix. For a matrix A with 
row vectors ai, . . . , a„i and a vector norm || • || in M"', the Hajnal diameter of A is defined 
by diam(A) ~ max \\ai — aj \\. The scramblingness 77 of a stochastic matrix A is defined 

as 

r]{A) = min \\ai A aj||i, (7) 

where A aj ~ [min(aii, aji), . . . , min(aim, aj„i)]. The stochastic matrix A is said to 
be scrambling if i]{A) > 0. The Hajnal inequality estimates the Hajnal diameter of the 
product of stochastic matrices. For two stochastic matrices A and B of the same order, the 
inequahty 

diam(AB) < (1 - ?7(A))diani(B) (8) 

holds for any matrix norm [30]. It can be seen from (8) that the diameter of the product 
AB is strictly less than that of i? if ^ is scrambling. 

The link between stochastic matrices and graphs is an essential feature of this paper A 
stochastic (or simply nonnegative) matrix A = [aij]™ =1 G M™^™ defines a graph Q = 
{V, £}, where V = {!,..., to} denotes the vertex set with to vertices and £ denotes the 
link set where there exists a directed link from vertex j to i, i.e., e{i,j) exists, if and 
only if Gij > 0. We denote this graph corresponding to the stochastic matrix A by G{A). 
For a directed link e{i,j), we say that j is the start of the link and i is the end of the 
link. The vertex i is said to be self-linked if e{i,i) exists, i.e., an > 0. ^7 is said to be 
a bigraph if the existences of e{i,j) and e{j,i) are equivalent. Otherwise, Q is said to a 
digraph. An L-length path in the graph denotes a vertex sequence (ui)^i satisfying that 
the link e{vi+i,Vi) exists for alH = 1, . . . , L — 1. The vertex i can access the vertex 
j, or equivalently, the vertex j is accessible from the vertex i, if there exists a path from 
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the vertex i to j. The graph Q has a spanning tree if there exists a vertex i which can 
access all other vertices, and the set of vertices that can access all other vertices is named 
the root set. The graph Q is said to be strongly connected if each vertex is a root. We 
refer interested readers to the book [11] for more details. Due to the relationship between 
nonnegative matrices and graphs, we can call on the properties of nonnegative matrices, or 
equivalently, those of their corresponding graphs. For example, the indecomposability of a 
nonnegative matrix A is equivalent to that Q{A) has a spanning tree, and the aperiodicity of 
a graph is associated with the aperiodicity of its corresponding matrix [15]. We say that Q 
is scrambling if for each pair of vertices i ^ j, there exists a vertex k such that both e{i, k) 
and e{j, k) exist, which can be seen to be equivalent to the definition of scramblingness 
for stochastic matrices. For two matrices A = [aij]fj^i, B = G M"'", we 

say A is an analog of B and write A B, in case that aij 7^ if and only if bij ^ 0, 
Vi, j = 1, . . . , n, that is, when their corresponding graphs are identical. 

Furthermore, for a nonnegative matrix A and a given d > 0, the d-matrix of A, denoted 
by A^, is defined as 



The S-graph of A is the directed graph corresponding to the (5-matrix of A. We denote by 
the neighborhood set of the vertex Vi in the (5-graph: ~ {vj : Aij > 5}. 

2.3. Convergence of products of stochastic matrices. Here, we provide the definition of 
consensus and synchronization of the system (5). Suppose the delays are bounded, namely, 
Tijicr'^) < TM for alH, J = 1, . . . , to and cr'° G fl. 

Definition 2.2. The multi-agent system is said to reach consensus via the algorithm (5) if 
for any essentially bounded random initial data G M™, t = 0,1,..., tm, (that is, 
is bounded with probability one), and almost every sequence {cr*}, there exists a number 
a G M such that lim = al with 1 = [1,1,...,!]^. The multi-agent system is said to 



synchronize via the algorithm (5) if for any initial essentially bounded random x G M™ and 
almost every sequence {cr*} , limt_i.oo \xi{t) — Xj{t)\ = 0, i,j = 1, . . . , to. In particular, 
if for any initial essentially bounded random a;" G M™, t ~ 0,1, . . . , tj^j, and almost every 
sequence, there exists a P-periodic trajectory s{t) (P independent of the initial values and 
the sequence) such that limj^oo \xi{t) — s{t)\ = holds for alH = 1, . . . , m, then the 
multi-agent system is said to synchronize to a P-periodic trajectory via the algorithm (5). 

In general, consensus can be regarded as a special case of synchronization, where the 
multi-agent system synchronizes at an equilibrium. As shown in Ref. [3], in the absence of 
delays, consensus and synchronization are equivalent w.r t. the product of infinite stochastic 
matrices; that is, whenever a system synchronizes, it also reaches consensus. However, 
we will show in the following sections that, under transmission delays, consensus and 
synchronization of the algorithm (5) are not equivalent. Thus, a system can synchronize 
without necessarily reaching consensus. 

Consider the model where the topologies are induced by a stochastic process: 



where {C*}ieN is a stochastic process with a probability distribution of the sequence P. The 
results of this paper are based on the following lemma, which is a consequence of Theorem 
2 in Ref. [3]. 





(9) 
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Lemma 2.3. Let •q{-) denote the scramblingness, as defined in (7). The multi-agent system 
via the algorithm (9) reaches consensus if and only if for P-almost every sequence there 
exist infinitely many disjoint integer intervals li = [a^, hi] such that 

oo / 

i—l ^ k—ai 

As a trivial extension to a set of SIA matrices, we have the next lemma on how to obtain 
scramblingness. 

Lemma 2.4. [34] Let Q C M™-™ be a set of SIA matrices. There exists an integer N such 
that any n-length matrix sequence with n > N picked from O; , , . . . , G" satisfies 



gM > 0. 

• k=l ^ 

The following result provides a relation between SIA matrices and spanning trees. 



Lemma 2.5. (Lemma 1 in Ref. [36] j If the graph corresponding to a stochastic matrix A 
has a spanning tree and a self-link at one of its root vertices, then A is SIA. 

3. Main results. We first consider the multi-agent network without transmission delays: 

n 

^E^^j^'^*)^?' * = I,---,™- (10) 

3 = 1 

The following theorem is the main tool for the proofs of the main results and it can be 
regarded as a realization of Lemma 2.3 and an extension from Ref. [18] without assuming 
self-links. 

Theorem 3.1. For the system ( 10), if there exist L G N and S > such that the 5-graph of 
the matrix product 

f n+L N 

J] G{a^)\r'\ (11) 

k=n+l ^ 

has a spanning tree and is aperiodic for all n G N almost surely, then the multi-agent 
system reaches a consensus. 

Theproofis given in Sec. 5.1. The main result of [18] can be regarded as aconsequence 
of Theorem 3.1, where each node in the graph was assumed to have a self-link. In the 
following, we first study the multi-agent systems with transmission delays such that each 
agent is linked to itself without delay and then investigate the general situation where delays 
may occur also on the self-links. Finally, we give a brief discussion on the consensus 
algorithms without self-links. All proofs in this section are placed in Sec. 5. 

3.1. Consensus and synchronization with transmission delays. Consider the consensus 
algorithm (6), which we rewrite in matrix form as 

^*^'=E^'('^*)^*"^ (12) 

T=Q 

where G(cr*) = [G[j (cr*)]"j^]^. We assume the following for the matrices G'^(-). 

A: Each G'^{a*), t G tjj_, is a measurable map from ^l to the set of nonnegative 
matrices with respect to T^. 
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X* ,a;*-i ]^ G M^x^^^^+i), we can write (12) as 



Letting 

where B(ct*) G R(rA/+i)xm,(r„+i)x,» jjjjg jhe form 



(13) 








... 
... 

••• /„ 



Thus, the consensus of (6) is equivalent to that of (13). As a default labeling, let us consider 
the corresponding graph 5(S(cr*)), which has {tm + l)m vertices, which we denote by 
{vi,j , i G Tm + 1, j G to}, where Vij corresponds to the ((i — 1)to+ j)th row (or column) 
of the matrix B{a* ). 

Theorem 3.2. Asswne the conditions A, and suppose there exist fi > 0, L G N, and S > 
such that GO(<t) > plmfor all a G and the S-graph of E{'^2^n+i G(cr*^)|J^"} has a 
spanning tree for all n £ N almost surely. Then the delayed multi-agent system ( 6) reaches 
consensus. 

The proof is given in Sec 5.2. In the case that the topological switching is deterministic, 
a similar result is obtained in the literature [24, 36]. 

Example 3.3. We give a simple example to illustrate Theorem 3.2. Consider a delayed 
multi-agent system on a network with 2 vertices and the maximum delay is 1. The system 
can be written as 

which can further be put into a form without delays = B{a*)y* with 

^ - I,n 

Let us consider the product of two matrices and B^: 



B' 




1 

V 






/ 



B' 



( 1/2 

1 

V 



In the absence of delays, they correspond to Gi = 



1/2 \ 



and G9 = 



1/2 1/2 
1 



One can see that the union of the graphs G{Gi) and G{G2) has spanning trees and self- 
connections. Then the proof of Theorem 3.2 says that for some integer L, the product of L 
successive matrices corresponds to a graph which has a spanning tree and a self-link on the 
root node. For example, we consider the following matrix product: 

/ 1/2 1/2 \ 

10 

1/2 1/2 ■ 

\ 10 / 

The corresponding graph has four vertices, which we label as wi 1, vi,2, V2.1, and V2,2 
following the scheme defined below Eq. (13). From Figure 1, it can be seen that the graph 



B^B^ 
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corresponding to B^B^ has spanning trees with vi.2 being the root vertex which has a 
self-link. So, by Theorem 3.2, the system reaches consensus. 




Figure 1. The graphs corresponding to the matrices-B^, B^, and the 
matrix product B^B^, respectively. 

In some cases delays occur at self-links, for example, when it takes time for each agent to 
process its own information. Suppose that the self-linking delay for each vertex is identical, 
that is, Tii = To > 0. We classify each integer t in the discrete-time set N (or the integer 
set Z) via mod (t + 1, tq + 1) as the quotient group of (Z + 1)/(to + 1). As a default 
set-up, we denote (i)To+i by (i). Let G"(-) = (')■ Foi" ^ simplified statement of 

the result, we provide the following condition B; 

B. 1 There exist an integer tq > and a number fi > such that G^" (cri ) > film far all 
(Ti £ fi; 

B.2 There exist ti, . . . ,tk excluding the integers in (0) with gcd{TQ + 1 , ti + 1 , . . . , tk + 
1) = P > 1 such that (cri) = 0/or all j ^ {ti, . . . , tk} and all cri £ and the 
S-matrix o/E{G'^'" (ct"+^)|J-'"} is nonzero far all 7i G N and k = 1, . . . , K almost 
surely. 

Theorem 3.4. Assume that the conditions A and B hold, and suppose there exist L G N 
and S > such that the S-graph o/E{^^''j[^^.^ G*'(cr'^)|J-'"} is strongly connected far 
all n Cz N almost surely. Then the system (6) synchronizes to a P-periodic trajectory. In 
particular, if P ~ 1, then (6) reaches consensus. 

The proof is given in Sec. 5.3. From this theorem, one can see that under self-linking 
delays, consensus is not equivalent to synchronization. In fact, the delays that occur on 
self-links are essential for the failure to reach consensus. 

Example 3.5. Theorem 3.4 demands that the J-graph corresponding to the matrix 
^{Y^^=n+i is strongly connected. This is stronger than the condition in The- 

orem 3.2, which demands that the corresponding graph has a spanning tree. We give an 
example to show that the strong connectivity is necessary for the reasoning in the proof. 
Consider a delayed multi-agent system on a network with two vertices and a maximum 
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Figure 2. The graph corresponding to the matrix product (14). 



delay of 3. Consider the form (13) and the matrix B{-). Suppose that the state space only 
contains one state cti as follows: 

" 1/3 1/3 1/3 ' 





1 
1 







1 







1 
1 




(14) 



Here, tq = 1. It is clear that the subgraph corresponding to each G\ ^ has spanning trees 
but is not strongly connected, and that there is a link between the subgraphs corresponding 
to (1) and (0). For the word (TiCTi • ■ • (J\a\, direct calculations show that the corresponding 
matrix product is an analog of the following matrix if the length of the word is sufficiently 
long: 

"110 10 10 
1 
01110101 
1 
110 10 10 
1 
01110101 
1 

The corresponding graph is shown in Figure 2, using the labehng scheme for the vertices as 
defined below Eq. (13). One can see that it does not have a spanning tree since the vertices 
v\^2 and V2,2 do not have incoming links other than self-links. In fact, the set of eigenvalues 
of the matrix B{p\) contains 1 and —1, which implies that (12) with B(a^) can not reach 
consensus even though the condition in Theorem 3.2 is satisfied. 

3.2. Consensus and synchronization without self-links. So far the stability result is 
based on the assumption that each agent takes its own state into considerations when up- 
dating. In other words, the coupling matrix has positive diagonals (possibly with delays). 
There also exist consensus algorithms that are realized by updating each agent's state via 
averaging its neighbor's states and possibly excluding its own [9]. In [5], it is shown that 
consensus can be reached in a static network if each agent can communicate with others by 
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a directed graph and the coupHng graph is aperiodic, which can be proved by nonnegative 
matrix theory [15]. In the following, we briefly discuss the general consensus algorithms 
in networks of stochastically switching topologies that do not necessarily have self-links 
for all vertices. 

When transmission delays occur, the general algorithm (6) can be regarded as increasing 
dimensions as in (13). Thus, one can similarly associate (13) with a new graph on m x 
{tm + 1) vertices {vij : i G tm + l,i G zn}, denoted by Q (•), where B{-) denotes the 
link set of C/ (■)' by which Vij corresponds to the (i — 1) x (tm + 1) + J column and row 
of B. Bp{(Ti) as the matrix corresponding the vertices {vij : i £ (p), j G m}. Based on 
theorem 3 . 1 , we have the following results, which can be proved similarly to Theorems 3.2 
and 3.4. 

Proposition 3.6. Assume A holds, and suppose there exist L G N and (5 > such that the 
S-graph o/IE{J^]!i^_|_]^ B{a'')\J-"^} has a spanning tree and self-link at one root vertex for 
all n G N almost surely. Then the algorithm ( 10) reaches consensus. 

In fact, under the stated conditions, each product IE{nfc=M+i B{a'^)\T'^} is SIA almost 
surely; so, this proposition is a direct consequence of Theorem 3.1. 

In the possible absence of self-links, the following is a consequence of Proposition 3.6. 

Proposition 3.7. Assume A and B.2 hold (B. 1 need not hold). Suppose there exist L £ N 
and 6 > such that the 6-graph ofM{YYk^^_^_i Bp{a'')\J-^} is strongly connected and has 
at least one self-link for all n £ N and p G P_ almost surely, where Bp is defined in the 
proof of Theorem 3.4, for example, ( 15) in Sec. 5.3. Then the algorithm (6) synchronizes to 
a P-periodic trajectory. In particular if P — 1, then the algorithm (6) reaches consensus. 

4. Applications. Adapted processes are rather general and include i.i.d processes and 
Markov chains as two special cases. Therefore, the results obtained above can be directly 
utilized to derive sufficient conditions for the cases where the topology switching and de- 
lays are i.i.d. or Markovian. 

First, by a standard construction as mentioned in Sec. 1, from the property of i.i.d. it 
follows that 'E{G{a''~^^)\J-''^} — E{G(o'*''+^)} is a constant stochastic matrix. Then, we 
have the following results. 

Corollary 4.1. Asswne that A holds and {cr*} is an i.i.d. process. Suppose there exist 
yU > 0, L G N, and S > such that C^i^a) > fj.I„i for all a £ fl and the S-graph of 
E{G((T^)} has a spanning tree. Then the delayed multi-agent system via algorithm (6) 
reaches consensus. 

Corollary 4.2. Assume that A and B hold and {cr*} is an i.i.d. process. Suppose there 
exist L G N and S > such that the S-graph ofE.{G^{a^)} is strongly connected for 
all n £ N almost surely. Then the system (6) synchronizes to a P-periodic trajectory. In 
particular, if P = 1, then (6) reaches consensus. 

Second, we consider the Markovian switching topologies, namely, the graph sequence 
is induced by a homogeneous Markov chain with a stationary distribution and the property 
of uniform ergodicity, which is defined as follows. 

Definition 4.3. [4] A Markov chain {tr*}, defined on {O, T}, with a stationary distribution 
TT and a transition probability T(a;, A) is called uniformly ergodic if 

\\T''{x, •) - 7r(-)|| ^ as A: ^ +oo, 
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where T''(-, •) denotes the A:-th iteration of the transition probabiHty T(-, •), for two proba- 
bility measures /i and v on {il, T)\, and \[l — v\\ — sup^gjr |^(^) — '^(-^)|- 

From the Markovian property, we have the following results. 

Corollary 4.4. Assume that A holds. Let {cr*} be an irreducible and aperiodic Markov 
chain with a unique invariant measure tt. Suppose {ct*} is uniformly ergodic and there exist 
/i > and 5 > such that G^{a) > iilm far all a ft and the S-graph of¥,-^{G{a^)} 
has a spanning tree. Then the delayed multi-agent system(6) reaches consensus. 

Proof. From the Markovian property, we have 

E{- J2 G{a')\T-} = E{- ^ G(a*)|a"}. 

t=n+l t=n+l 

If {ct*} is uniformly ergodic, then 
lim E{i 'j2 G(a*)|a"}= lim yE / G(y)rK,dy) = 

l=n+l i=l 

Since the convergence is uniform, there exits L such that the (5/2-graph corresponding to 
E{(l/L) X]"=!ri+i G(ct*)|J-'"} has a spanning tree almost surely. From Theorem 3.2, the 
conclusion can be derived. □ 

Corollary 4.5. Assume that A and B hold, and let {cr*} be an irreducible and aperiodic 
Markov chain with a unique invariant measure tt. Suppose that {cr*} is uniformly ergodic 
and there exists S > such that the S-graph o/E^{G°(cr^)} is strongly connected. Then 
the system (6) synchronizes to a P -periodic trajectory. In particular, if P = 1, then (6) 
reaches consensus. 

These corollaries can be proved directly from Theorems 3.4 in the same way as Corol- 
lary 4.4. It can be seen that the a homogeneous Markov chain with finite state space and 
unique invariant distribution is uniformly ergodic. Hence, the results of Corollaries 4.4 and 
4.5 hold for this scenario. 



/ G{yMdy)^E4G{a% 
Jn 



5. Proofs of the main results. In the following, the coupling matrix B{-) in the delayed 
system (13) is written in the following block form: 

Si.i(f7*) Si.2(a*) ••• Si.,„+i(a*) 

S2.l(fT*) B2,2(CT*) ••• B2,,„ + i(a*) 



i3(a*) 



B 



(a*) B 



(<7*) 



B 



tj\/ + 1,ta/ + 1 



(a*) 



p ( Tj./ + 1 ) m , (ta J + 1 ) ?ri 



with Bij (cr*) S M™'™, i, j G tj\/ + 1. For two index sets / and J, we denote by [B{a*)]j,j 
the sub-matrix of B{a*) with row index set / and column index set J. For an n-length 
word cr = {cr^)^^i in the stochastic process, we use B{a) to represent the matrix product 
nr=i -^i'^^)- ^^^^ structure of the matrix B{a*) has the following proper- 

ties: (1). Each Bi^i-i = I„i for all i > 2; (2). Bi j = for alH > 2 and j ^ i — 1. These 
properties are essential for the following proofs. 

As the same way defined below Eq. (13), let us consider the corresponding graph 
Q{B{a*)), which has (tm + 1)™ vertices, which we denote by {vij, i G tm + 1, ,j G m}, 
where Vij corresponds to the {i — l)m + j row of the matrix B{a). 



CONSENSUS AND SYNCHRONIZATION IN NETWORKS 



13 



We denote the following finitely generated periodic group: 

K 

(ii, 12, . . . , in) -J ■■= {p. P = X! ^^P^ ^J'- 

l=k 

If these numbers are be picked in a finite integer set, for instance, {1, . . . ,tm + 1} in 
the present paper, then (ii, 12, • • • , iK)j denotes the set (ii, 12, • • • , ix)] Pi '^m + 1 unless 
specified otherwise. As a default setup, (i) denotes (j)ro+i where tq is the self-linking 
delay as in (12). We will sometimes be interested in whether an element in a matrix is zero 
or not, regardless of its actual value. 

5.1. Proof of Theorem 3.1. From the condition in this theorem, we can see that the 5- 
matrix of E{nl~n+i G((t'')| J^"} is SIA for all n e N. Lemma 2.4 states that there exists 
e N such that the product of any A^ SIA matrices in M'"-'" is scrambling. Note that 

n+NL ~. f f f n+NL 



^t=n+l J L L ^ tL=n+{N-l)L + l ^ 

" ' 'N-1)L , n+L s 

f(Ar-2)L + l ^ ti=Ti+l ^ 



tL-l=n+{N-2)L + l 

since {J^*} is a filtration. This implies that there exists a positive constant 61 < 5^ such 
that the (5i -graph of E{n"i!+i G((t*)|J-""} is scrambling. So, from Lemma 3.12 in Ref. 
[18], there exist d' > and Mi e N such that 



^ ^ t=ri+l 



Let Ck = Ilt^feMfTVL+i ^i'^*)- '-^ conclude that for almost every sequence of {a*}, 
it holds that 

lim Vp/nfCfc) > S'\T''^^\ > Mm K xS' ^ +00. 

k=l ^ ' 

From Lemma 2.1, we can conclude that the events {i^iCk) > S'}, k = 1,2,..., occur 
infinitely often almost surely. Therefore, we can complete the proof directly from Lemma 
2.3. 

5.2. Proof of Theorem 3.2. The proof of this theoremis based on the structural character- 
istics of the product of matrices B{-). We denote by [B{-)]ij the M™'™ sub-matrix of B{a) 
in the position We first show by the following lemma that the graph corresponding to 
the product of more than tm + 1 successive matrices B{a* ), as defined by (13), has a span- 
ning tree and self-link at one root vertex. Thus, we can prove Theorem 3.2 by employing 
Theorem 3.1. 

Lemma 5.1. Under the conditions in Theorem 3.2, for any n-length word a = (cOT^i 
with n > T]\j + 1, there exists fii > such that 

(ii). j:-=i'[Bi<j)]i,, > ^^^J:■=^'El=lGH<y'')■ 
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Proof. We choose < /ii < /j,, where ji is defined in Theorem 3.2. (i). For a word 

cr = (o'i)f^i with n>TM + 1, 



Jlr.,1 

' n— i+1 



[B{<J)U = J2 [SK)].,n[5(^n-i)]n,..---[S(ai)] 

ii ,...,in 



k=l 

since fc+j— Ti,fc+i — ri — 1 — Im for all fc > 71 — i + 2 and [i?(t37)]i i > jllm > jJ^lIn 

for all w £ ri. 

(ii). Let i G Tyj/ + 1 and to G 2i- If ^ J- we have 

Y,[B{a)],.i - ^ [i3K)]i,.,[i?K_i)],;,,,,.-.[i?(ai)].„,z 

> ( n [SK)]ia)[SK,)]ij( 11 [S(aO]i-to+,J-to+^-i) 

-to-J + 1 

n 

since > /ii/„i, = /„i for all I > - j + 2 for all 

tu G 51; whereas if j > we similarly have 

/ n \ /to-1 

Summing the right-hand side of the above inequality with respect to to and jproves (ii). □ 

Proof of Theorem 3.2. Let us consider the /i" -graph of B{(t) for all cr ~ 
with 71 > TM + 1, as defined in Lemma 5.L The item (i) in Lemma 5.1 indicates that 
for each vertex vij with i > 2 and j G m, there exist a path from vertex vij to Vij: 

(wij,W2J,...,W»j). 

From item (ii) in Lemma 5. 1 and the conditions in Theorem 3.2, one can see that there 
exits 5 > and L G N such that the J-graph of Y.iMU'^=n+i ■S(o-*)| has span- 

ning trees and self-links. Let Q be the random variable corresponding to the (5-graph of 
^{nrif+i S((7*)|J-""} and Q' be the random variable corresponding to the (5-graph of 
SJII^inrii+i Then, for almost every graph Q\ there exists an index 

jo G m such that for any j, there exists a path (jo, Ji, • ■ • ,jK-i,j) to access j. This im- 
plies that for almost every graph Q, there exists a path from vijg to vij. Thus, vij„ can 
access all vertices Vij, i ~ 1, . . . , tm + 1, since vij can access all Vij for tm + 1 > i >2 
by a directed link and wij^ and has self-link, noting that G"(-) has positive diagonals. 
Therefore, for almost every graph Q, it has a spanning tree and the vertex wi j„ is one of 
the roots. From Lemma 2.5, one can see that IE{nr=rf+i S((7*)| J""} is SIA almost surely. 
According to Theorem 3.1, the system (10) reaches consensus. This proves the theorem. 
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5.3. Proof of Theorem 3.4. Outline of the proof: For a better understanding of the proof, 
we first give the following sketch. We start the proof by defining a permutation matrix 
Q e corresponding to the permutation sequence from (1,2..., tm + 1) to 

((1), (2), ... , (P)). Then we show by the lemma that follows that the matrix B{a^) can be 
transformed into the following form: 

" Biia') •••0 

B2(tT*) ••• 



[Q(E>UB{a')[Q(g>W = 











(15) 



(16) 

This system can be divided 
(17) 



where (g) stands for the Kronecker product and Bp{(T*-) = _B((^p^|j,^ ^^p^i^,^ (tr*). By the 
permutation Q, we can rewrite the coupled system (5) as 

where y* = [Q ® /,„]y* and I3{-) = [Q® IJ\B{-)[Q ® /„ 
into P subsystem as 

where y* corresponds to [?/*]{(p)|p). So, it is sufficient to prove the following claim to 
complete this proof from Lemma 3.1: 

Claim 1: For each p E P, there exists 6' > and L G N such that the i5'-graph of the 
matrix 

e{ n Bp{a')\F^\ (18) 

has a spanning tree for all ?? G N almost surely. 

The proof of this theorem is also based on the structural characteristics of the product 
of matrices B{-). By the lemmas below, we are to show the permutation form (15) can be 
guaranteed. 

Lemma 5.2. Under the conditions of Theorem 3.4, for any (tq + \)-length word a = 
((Tfe)^^ , there exists some pi > such that the following hold: 

(i) . [B{a)]u > p]"^^I,nforallie tq + 1 ; 

(ii) . [B{a)]j_j_(^ro+i) > Imforallj > Tq + 2; 

(Hi)- Y.i^(i-j)[B{cj)]r„+2-u > p]°+^G°ia,)forallj E TQ + l ; 

C'W- J2ie{i+(T+i))[Bi<j)]i.i > ^i'["^^[B{aro+2-i)]i,T+ifor all i G tq + 1 and t G tm- 



Proof. We choose < pi < p. (i). For any i G tq + 1 , we have 



-( n [B{<^p)]p+i-l-To,p+i-2-Taj[B{<Tro-.t+2)]l,T„ + l 
p=To+3—i 
-To-i+l N 

Y[ [B{aq)]g+i^g+i_i j > film > fJ'l°^^Im 
q=l ' 

since [i3(n7)]i+i,i = /,„ and \B(w)\i^toj^i > M-^m for all tu G and i G tj\/. 
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(ii) . For any j > tq + 2, we have 

ii,...,To 

TO + 1 

> H [B{<Jk)]k+3-T„-l,k+j-To~2 = Im 
fe=l 

since [B{'iu)]i+i_i = I„i for all i > 2 and G fi. 

(iii) . For any i G tq + 1, we have 



> 



'W[B{ara-i+k+l)]k,k-l ) [-B((T^o_i+2)]l,(fc+l)(ro + l) 
fc=2 
(A;+1)(to + 1) 



]^ [-B(cr;_A.(T.„ + i)_i)](^;_l j > [S(<TT-„+2-i)]l,(fe+l)(To + l) 

i=i+(n)+i)(fc+i)-n) ^ 
for all fc > 0. Summing the right-hand side with respect to fc and letting j = tq + 2 — i, we 
have Eie(i-j)[^('^)]ro+2-j;; > E;e(ro+i) 
(iv). Let j = To + 2 - i. If j > r, 

/ To + l s 

X![-^'^^)]n)+2-i.^o+2-J + (r + l) + (ro + l)fe > ( [^(^p)]p-J + l:P-J ) 

fc ^P=j+1 ^ 

[B((Tj)]i^r+l ( W [B((Tq)],+^+2-j,g+T + l-j j [-B((Tj_r-l)]l,ro + l 

q=j~-r 

.j-r-2 . 

H [-S(o-;)]/+r+ro+3-j,;+r+ro+2-i j > Ai[S(o-j)]i,^+i; 

;=i ^ 
whereas if j < r, 

/ ro + l X 

Y[Bi(j)]ro+2-j,To+2~j + {T+l) + {ro + l)k > ( Yl [^('^p)]p-i+l.P-J ) (^J )] I'^+l 
fc \p=j+l ^ 

Y\_[B{aq)]g+r+2-j,q+T+l-j j > [B{aj)]i^r+1- 
- q=l ^ 

These calculations complete the proof of the lemma. □ 

Lemma 5.3. Under the conditions of Theorem 3.4, consider an L(tq + l)-length word 
a = ((Ti, . . . , (7l), where each ai = (fi.OI^l^ (''o + i)-length word. If L > tm + 1, 
then there exists /ii > iMc/i that 

(i). [B{d)]jA > ti'-^"^^^^Imforall j G {i) and i G tq + 1 ; 

f/nj. Eje{j+r+i)[-S('5')]».j > E;e(r+i>[-S(^ro+2-»)]i,;/orfl// i G tq + 1 and t G 

fh'j. //'t' ;i iMc/z that t' + 1 ^ (tq + 1, ti + 1, . . . , T/f + 1) anii [B(<7i)]i (^^,_^i^ = Ofor 
all (Ti G r2, f/xen [-B((T)],j ^i+^/^i^ = Ofor all i > 1. 
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Proof. We pick some < yU. (i). For j < tq + 1, the proof is similar to the proof of item 
(i) of Lemma 5.2. For j > tq + 2, we have 

l — li p— 1 

where li = L + 1 — {j — i)/ (tq + 1) is an integer (noting j G (i)), since 
[S(CT;)]j_(L-()(ro+i) j-(L-i+i)(ro+i) > Irn holds here, as mentioned in Lemma 5.2 (ii). 

The items (ii) and (iii) can be proved by similar arguments as in the proof of items 
(iii) and (iv) of Lemma 5.2. It remains to prove item (iv). In the following, we will prove a 
slightly more general result, namely that [B{<T)]i /^i_^_^,^i'j = for all words cr having length 
L E {tq + 1). Let (T = {(Ji)f^i be an arbitrary L-length word. We calculate [B{<T)]i,j with 
j e (i + t' + 1) as a sum of several matrix product terms: 

ii,...,it-i 

Since any zero factor yields zero product, we avoid zero factors in the calculations. That 
is, in the expression above, only factors of the form [S((T;)]j+i ^ and [B{ai)]i,j can occur 
where j E {i + t' + 1) and t' + 1 ^ {tq + 1,ti + 1, . . . ,tk + I) ■ Thus, letting ji = i, we 
have 



fry / ^p=i jp 

iB{<y)k,- E H n [^K)]i:u^.+^--^^i:U.^.+^--^-i 

-| ,^^Y^p=i3p 
^ kv+i = l 

where each jp e (tq + 1, ri + 1, . . . , r/f + 1). Suppose that the matrix product is nonzero. 
Then j = Y.l=i Jp - i.e., ((i + r' + 1) - h ^ ^)) = 0' which implies (r' + 

1 - YZ=2 jp + i> = 0. This means that t' + 1 e (tq + 1, Tfc + 1 : A: = 1, . . . , A'), 
which contradicts the condition r' + 1 ^ (tq + 1, + 1 : fc = 1, . . . , A'). The lemma is 
proved. □ 

Proof of Theorem 3.4. Consider the graph Q^{(j^) — {V, f (cr*)} on [tm + 1)™ vertices 
corresponding to the 5-graph of the matrix B{a^ ) as defined at the beginning of this section. 
For L G N as fixed in the main condition of Theorem 3.4 and an arbitrary fixed m £ N, let 
® = E{nr=™+i B{a*^)\r'} and be the random variable picked in the 5-graphs of B. 

First, we divide the graph into tm + 1 subgraphs: Qf, = {Vfc, ffc(cr*)}, fc £ tj\/ + 1, 
where Vk = {vk,i ■ i G Hi} corresponds to the rows or columns of Mk.k and the vertex 
Vk.i corresponds the i-th row or column of the matrix Mk.k- Then, integrate the subgraphs 
{Gi}l"t' into TO + 1 subgraphs: G'f = {V/,£:/}, I G rp + 1 , where V/ = Ufce(0 ^k, 
Z G Tq + 1 and 5/ corresponds to the intra-links in V/. Let £ii,i2 denote the inter-links from 
the subgraph of V,'^ to the subgraph V/^ . Lemma 5.3 (i) implies that for each Z G tq + 1, 

there must exist a link from vi i to v^ i in the subgraph (•), for each vertex Vk^i G Vk 
with fc > I and fc G (l). Similarly to the the proof of Theorem 3.2, the main condition 
of Theorem 3.4 and items (ii) and (iii) in Lemma 5.3 imply that there exist > and 
A G N such that the subgraph Q'^i^ is strongly connected, consequently having a spanning 
tree, and each vertex in Vi is one of the roots in C/'^^ and has a self-link almost surely for 
alU G To + L 
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Second, according to gcd(ro + 1, Tfe + 1 : fee K_) = P, we integrate the subgraphs 
g'^i' for all I e To + 1 , into P subgraphs, denoted by = {Dp, £p}, p G P by Vp = 
{V'j : £j.p ^ 0}. The items (ii) and (iii) in Lemma 5.3 and the second item in condition 
B indicate that the (5i -matrix of J2j£{Tk+i- k=o i k) ^i-J+j positive for all I £ tq + 1 . 
This implies that there exists at least one link from G'^i\.j to Q'f^ and this link end in V/. 
So, in the graph Q'^'^ , the root vertex in G'f\.j can reach all vertices in Q'^i^ since each 
vertex in V; is a root vertex in fj'f ^ . This leads to the conclusion that Vj C V; provided 
j — Z G (t/j + 1 : fc = 0, 1, . . . , K). Also, we can conclude that each root vertex in g'i+/^ 
can reach all vertices in t/'f\ by item (i) in Lemma 5.3. Therefore, we can conclude that 
~ U/e(p)p ^^'^ each Qp has a spanning tree almost surely. This proves Claim I. 
Moreover, there exists a vertex with self-link in Vi, i G tq + 1 and i G {p)p, as one of 
its roots, in Q^^ . So, according to the arbitrariness of integer n, we can conclude that the 
(5i-graph of E{nr=iVi i?p((T*)| J""} is SIA almost surely for all n G N. 

Finally, according to the second item in condition B and the (iv) item in Lemma 5.3, 
one can conclude that there are no links between the graph for different p E P_ for any 
5 > 0. So, by a permutation matrix Q corresponding to the permutation sequence from 
(1, 2 . . . , TM + 1) to ((1), (2), . . . , (P)), [Q ® /™]B(cr*)[Q ® /,„]T has the form (15). 

By Theorem 3.1, we can conclude that (17) reaches consensus for all p = 1, . . . , P, but 
converges to different values except for initial values in a set of Lebesgue measure zero. 
Therefore, a;* can synchronize and converge to a P-periodic trajectory. This completes the 
proof of Theorem 3.4. 

6. Conclusions. In this paper we have studied the convergence of the consensus algorithm 
in multi-agent systems with stochastically switching topologies and time delays. We have 
shown that consensus can be obtained if the graph corresponding to the conditional ex- 
pectations of the coupling matrix product in consecutive times has spanning trees almost 
surely and self-links are possible. With multiple delays, if self-links always exist and are 
instantaneous (undelayed), then consensus can be guaranteed for arbitrary bounded delays. 
Moreover, when the self-links are also delayed, we have shown the phenomenon that the 
algorithm may not reach consensus but instead may synchronize to a periodic trajectory ac- 
cording to the delay patterns. Finally, we have briefly studied consensus algorithms without 
self-links. We have presented several results for i.i.d. and Markovian switching topologies 
as special cases. 
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